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ABSTRACT 

Real  time  computer  graphics  are  limited  in  that  they 
can  only  be  displayed  on  projection  screens  and  monitors. 
Monitors  and  projection  screens  cannot  be  used  in  live 
fire  training  or  scenarios  in  which  the  displays  could  be 
physically  damaged  by  trainees.  To  address  this  issue,  we 
have  developed  projection  systems  using  computer  vision 
based  color  correction  and  image  processing  to  project 
onto  non-ideal  surfaces  such  as  painted  walls,  cinder 
blocks,  and  concrete  floors.  These  projector-camera 
systems  effectively  paint  the  real  world  with  digital  light. 
Any  surface  can  become  an  interactive  projection  screen 
allowing  unprepared  spaces  to  be  transformed  into  an 
immersive  environment.  Virtual  bullet  holes,  charring, 
and  cracks  can  be  added  to  real  doors,  walls,  tables, 
chairs,  cabinets,  and  windows.  Distortion  correction 
algorithms  allow  positioning  of  projection  devices  out  of 
the  field  of  view  of  trainees  and  their  weapons.  This 
paper  describes  our  motivation  and  approach  for 
implementing  projector-camera  systems  for  use  within  the 
FlatWorld  wide  area  mixed  reality  system. 

1.  INTRODUCTION 

Projection  technology  is  widely  used  for  immersive 
virtual  reality  applications.  In  comparison  to  head 
mounted  displays,  projection  systems  facilitate  free 
movement  of  participants  and  can  be  tightly  integrated 
with  the  physical  environment. 

1.1.  Rear  projection  immersive  virtual  reality 

There  are  two  basic  approaches  used  for  projection 
based  virtual  reality:  projection  from  behind  the  screen 
and  projection  from  the  front.  An  example  of  the  latter  is 
the  “Being  There”  project  [Low  et  al.,  2001]  at  the 
University  of  North  Carolina  at  Chapel  Hill.  In  this 
system,  walls  of  white  styrofoam  blocks  are  arranged  to 
reproduce  the  basic  layout  of  a  room.  Imagery  is  front 
projected  onto  the  styrofoam  blocks  making  these 
surfaces  appear  as  textured  walls  with  virtual  windows 
and  doors. 

Front-projection  systems  are  compact,  but  there  are 
drawbacks:  shadows  caused  by  people  walking  in  front  of 
projectors  and  difficulties  preserving  light  polarization 
needed  for  passive  stereoscopic  display. 


Most  rear  projection  based  virtual  reality  systems  are 
derived  from  the  CAVE  [Cruz-Neira,  Sandin  and  DeFanti, 
1993].  The  CAVE  is  a  rear  projected  cube,  where  3  to  6 
sides  are  screens,  creating  a  highly  immersive  virtual 
environment. 

The  FlatWorld  project  at  the  University  of  Southern 
California  Institute  for  Creative  Technologies  [Pair  et  al., 
2003  and  Treskunov,  Pair  and  Swartout,  2004]  creates  a 
mixed  reality  environment,  combining  stereoscopic  rear- 
projection  with  physical  props  tightly  integrating  virtual 
and  real  scene  elements. 

Real  scene  elements  such  as  walls  and  furniture 
introduced  by  the  Flatworld  concept  could  be  further 
enhanced  by  incorporating  front-projection  technology. 
For  example,  the  Shader  Lamp  system  (Raskar  et  al, 
2001)  suggests  that  projecting  onto  color  neutral  real 
objects  can  reproduce  different  “appearances,  including 
alternate  shading,  lighting,  and  even  animation.” 

Our  prototype  projector-camera  system  described 
later  in  the  paper,  utilizes  front-projection  for  simulating 
bullet  damage  on  a  real  wall. 

1.2.  Limitations  of  front  projection  technology 

As  mentioned  previously,  two  drawbacks  of  front 
projection  are  shadows  and  the  loss  of  light  polarization. 
In  FlatWorld,  we  can  alleviate  the  shadow  problem  either 
by  placing  projectors  and  props  such  that  users  do  not 
intersect  projection  beams  or  by  using  multiple  projectors. 
The  loss  of  stereo  polarization  is  not  critically  important 
since  we  plan  to  use  front  projection  for  simulating 
damage  effects  that  appear  realistic  even  though  the 
imagery  is  not  stereoscopic.  Our  approach  is  challenging 
in  that  we  plan  to  project  onto  unprepared,  arbitrary 
surfaces  as  opposed  to  specialized  screens  or  neutral 
surfaces  as  in  the  “Being  There”  or  “Shader  Lamp” 
approaches. 

By  avoiding  the  shadow  and  polarization  issues,  new 
problems  are  introduced  in  our  system.  We  placed  the 
video  projector  in  an  out-of-sight  location  at  an  oblique 
angle  which  creates  geometric  distortion  in  the  projected 
image.  Furthermore,  using  regular  objects  such  as  walls 
and  tables  as  projection  targets  is  problematic  since  they 
do  not  have  the  properties  of  ideal  flat,  white  matte 
screens. 
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Consequently,  there  is  a  need  for  a  geometry  and 
radiometric  compensation  step  before  projecting. 
Fortunately,  the  computer  vision  research  and 
development  community  provides  answers  we  can 
employ. 

2.  RELATED  COMPUTER  VISION  WORK 

Ramesh  Raskar  proposed  a  rendering  technique  to 
display  perceptively  correct  images  when  a  projector  is 
oblique  with  respect  to  a  planar  display  surface  [Raskar, 
1999  and  Raskar,  2000].  His  proposed  method  can  be 
integrated  into  a  graphics  engine,  incurring  no  additional 
cost  during  real-time  rendering.  If  the  surface  is  not 
planar,  two  pass  rendering  may  be  employed  [Raskar  et 
al,  1998].  During  the  first  pass  the  "desired  image  is 
stored  as  a  texture  map.  In  the  second  pass  the  texture  is 
effectively  projected  from  the  user’s  viewpoint  onto  the 
polygonal  model  of  the  display  surface." 

The  CAVE  Lab  at  Columbia  University  developed  an 
algorithm  for  color  correction  when  projecting  onto 
lambertian  colored  surfaces  without  inter-reflections 
[Nayar  et  al.,  2003  and  Grossberg  et  al.,  2004].  In  later 
work  [Fujii,  Grossberg  and  Nayar,  2005],  they  adapted  the 
method  to  dynamically  handle  changing  lighting 
conditions  to  a  certain  extent. 

Modern  projectors  have  limited  dynamic  range.  As  a 
result,  applying  global  color  compensation  techniques 
creates  clipping  problems  and  low  contrast  images.  Work 
has  been  conducted  to  overcome  this  problem  by 
exploiting  human  visual  perception  properties  for  several 
projectors  [Majumder  and  Stevens,  2005]  and  for  dynamic 
display  conditions  [Grundhfer  and  Bimber,  2006]. 

In  practical  situations,  complex  lighting  effects  need 
to  be  taken  into  account.  The  augmented  reality  group  at 
Bauhaus  University  recently  proposed  [Bimber  et  al., 
2006]  a  method  for  compensating  for  indirect  scattering  in 
real-time  using  reverse  radiosity.  Such  scattering 
compensation  is  essential  when  projection  screens  are 
concavely  shaped. 

More  complex  effects  need  to  be  addressed  in 
situations  in  which  subsurface  scattering  is  important  such 
as  projecting  onto  marble.  The  full  interaction  between 
the  projected  and  observed  image  is  described  by  the  light 
transport  matrix.  An  efficient  method  for  capturing  this 
matrix  has  been  proposed  by  a  recent  paper  from  the 
Stanford  Computer  Graphics  Lab  [Sen  et  al.,  2005].  To 
apply  light  transport  based  compensation,  one  needs  to 
either  solve  a  system  of  linear  equations  or  apply  the 
pseudo-inverse  of  the  light  transport  matrix  in  real-time  as 
has  been  done  by  [Wetzstein  and  Bimber,  2006]  using  a 


graphics  card's  GPU.  This  technique  is  the  most  general 
approach  for  a  single  projector  case. 

It  is  also  possible  to  use  multiple  projectors  to 
increase  depth  of  field  [Bimber  and  Emmerling,  2006(2)] 
or  to  eliminate  shadows  [Cham  et  al.,  2003]. 

3.  FLATWORLD  INTEGRATION 

In  early  2006  we  began  prototyping  the  use  of  front- 
projection  and  projector-camera  systems  in  the  FlatWorld 
wide  area  mixed  reality  environment.  Our  initial 
demonstration  aimed  to  simulate  AK-47  bullet  impacts  on 
an  unprepared  real  wall  of  the  FlatWorld  environment. 
The  bullets  would  be  fired  by  a  virtual  stereoscopic  rear- 
projected  insurgent  firing  into  the  room  (see  Fig.  1).  To 
avoid  shadows  from  participants,  we  planned  to  place  a 
projector  above  the  ceiling  at  a  sharp  angle  with  respect  to 
the  wall  projection  target  (see  Fig.  2).  Computer  vision 
based  geometry  correction  would  be  used  to  compensate 
for  the  non  ideal  projector  alignment. 


Fig.  1:  Conceptual  rendering  of  a  virtual  insurgent  firing 
virtual  bullet  holes  into  a  real  wall. 


Fig.  2:  Projector  placed  above  the  ceiling. 


3.1.  Calibration  and  integration  with  rendering 
pipeline 

Our  geometry  compensation  technique  is  based  on 
Ramesh  Raskar’s  work  [Raskar,  1999  and  Raskar,  2000]. 

In  our  approach,  the  system  is  calibrated  by 
projecting  a  chessboard  pattern  and  photographing  it  with 
a  camera  placed  in  an  ideal  projector  position  in  front  of 
the  projected  area  (Fig.  3).  During  this  stage,  we  decrease 
noise  by  applying  a  temporal  average  to  a  sequence  of  100 
captured  distortion  corrected  images.  The  OpenCV  library 
[OpenCV]  was  used  for  camera  calibration.  To  correct 
for  distortion,  chessboard  comers  are  extracted  with  sub¬ 
pixel  accuracy  and  a  3x3  homography  matrix  H  is 
calculated  which  transforms  normalized  2D  coordinates  of 
the  captured  chessboard  corners  into  projected  corners. 
Comer  coordinates  are  normalized  to  be  in  the  [-1;  +1] 
range  to  eliminate  the  different  spatial  resolutions  of  the 
projector  and  camera. 


Fig.  3:  Calibration  pattern 


The  matrix  H  was  loaded  during  the  initialization  of 
the  rendering  application  [Treskunov,  Pair,  and  Swartout, 
2004].  It  was  transposed  to  take  into  account  different 
matrix  conventions  between  OpenCV  and  DirectX.  Next, 
the  4x4  projection  matrix  A  was  constmcted  as  in 
[Raskar,  1999  and  Raskar,  2000]. 


hn  h2l  0  h31 

^22  0  h32 

0  0  1  —  | h3l  |  —  | h32 1  0 

h\3  h23  0  1 


Inside  the  rendering  setup  call,  the  projection  matrix 
P  was  calculated  as  usual  given  camera  and  viewport 
information.  It  was  then  modified  by  the  stored  matrix  A. 


Here,  multiplication  order  is  reversed  to  take  into 
account  transposition.  The  p' 34  element  was  reset  to 
equal  1  after  multiplication  for  the  matrix  to  be  “W- 
friendly”. 

After  this  setup  process,  the  rendering  loop  mns 
without  additional  overhead. 

3.2.  The  advantages  of  physical  props 

As  mentioned  earlier,  the  use  of  physical  props  in  the 
FlatWorld  system  avoids  many  of  the  challenges  that  exist 
in  rear  projection  systems  like  the  CAVE.  For  example, 
a  vertical  column  placed  at  the  corner  of  two  screens 
masks  rendering  latency  between  two  projection  channels. 
Rubble  placed  on  the  floor  blocks  a  user  access  to  areas 
where  rendering  artifacts  appear  when  a  tracked  user  is 
too  close  to  the  screen.  The  elements  are  placed  in  such  a 
way  that  they  do  not  harm  the  overall  user  experience. 

In  a  similar  fashion  we  resolved  the  problems 
introduced  by  DLP  projector  black  levels.  Because  DLP 
projectors  cannot  project  pure  black,  the  projected  area  is 
clearly  visible  and  appears  artificial.  To  alleviate  this 
problem,  we  strategically  placed  a  spot  light  to  mask  the 
boundary  of  the  projected  area. 

3.3.  Initial  demonstration 

Our  prototype  scenario,  completed  in  August  2006,  is 
seen  in  Figure  4  and  Figure  5.  After  a  virtual  insurgent 
fires  rounds  into  the  room,  a  damage  animation  is 
displayed  on  the  opposite  wall,  creating  highly  convincing 
damage  effects. 


Fig.  4:  Virtual  insurgent  firing  toward  a  real  wall  in  the 
FlatWorld  one  room  prototype. 


P'=P*A 


Fig.  5:  Realistic  computer  generated  bullet  holes  and 
debris  appear  on  the  opposite  real  wall. 


CONCLUSIONS 

Our  ongoing  projector-camera  system  research  and 
development  work  will  allow  us  to  transform  any  solid 
surface  into  a  projection  screen.  This  capability  allows  a 
virtual  insurgent  presented  on  a  digital  flat  to  be  able  to 
fire  a  weapon  and  destroy  a  real  wall  in  the  room.  An 
angry  virtual  civilian  could  throw  virtual  rocks  that  crack 
real  tables  and  chairs.  Our  work  also  facilitates  virtual 
humans  capable  of  casting  projector  generated  shadows 
into  the  real  world.  Training  scenarios  utilizing  our  work 
will  be  more  stressful  and  convincing,  better  preparing  the 
soldier  for  similar  situations  on  the  real  battlefield. 
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